Apparatus and method for increasing a number of points in a point cloud

专利摘要:
FILL POINTS IN A POINT CLOUD. The present invention relates to an apparatus, system, and method for increasing points in a point cloud (132). In an illustrative embodiment, a two-dimensional image (121) of a scene (110) and point cloud (132) of the scene (110) are received. At least a portion of the points in the point cloud (132) are mapped to the two-dimensional image (121) to form transformed points (146). A merged data network (150) is created using the two-dimensional image (121) and the transformed points (146). New points (164) for the point cloud (132) are identified using the merged data network (150). The new points (164) are added to the point cloud (132) to form a new point cloud (162).
公开号:BR102013028714B1
申请号:R102013028714-8
申请日:2013-11-07
公开日:2022-02-01
发明作者:Terrell Nathan Mundhenk；Yuri Owechko；Kyungnam Kim
申请人:The Boeing Company；
IPC主号:

专利说明:

FUNDAMENTAL INFORMATION FIELD
[0001] The present description relates generally to point clouds and specifically to increasing the resolution of point clouds. Even more specifically, the present disclosure relates to a system and method for increasing the number of points in a point cloud to increase the resolution of the point cloud. BACKGROUND
[0002] A point cloud is a collection of points in a three-dimensional (3D) coordinate system that describes a three-dimensional scene. Typically, points in a point cloud represent the outer surface of objects. A light detection and range (LIDAR) system is an example of a type of sensor system capable of generating a point cloud. Point clouds can also be generated using, for example, stereo camera systems, mobile laser imaging systems, and other types of sensor systems.
[0003] Point clouds can be used to perform various operations such as, for example, object identification, object classification, scene visualization, segmentation, two-dimensional image data enhancement, and/or other types of operations. The performance level with which these operations are performed using a point cloud may depend on the resolution of that point cloud.
[0004] As used herein, the "resolution" of a point cloud can be the level of detail with which features in the scene captured by the point cloud can be discerned within the point cloud. The resolution of a point cloud may depend on the number of points in the point cloud and/or the point density of points in one or more portions of the point cloud. As used herein, "dot density" is a measure of the number of dots per unit volume. A portion of a point cloud that has a higher density than another portion of the point cloud may be less sparse than the other portion.
[0005] In some situations, an object identification, object classification, segmentation, and/or visualization of a scene using a sparse point cloud can generate inaccurate results. For example, a point cloud may be insufficiently dense to correctly identify or classify an object.
[0006] Some currently available solutions for increasing the number of points in a point cloud include making assumptions about objects in the scene. For example, assumptions can be made about the shape of an object in the scene and new points can be added to the point cloud based on these assumptions. However, with these types of solutions, the locations in the three-dimensional reference coordinate system to which new points are added may be less accurate than desired.
[0007] Also, some currently available solutions may be unable to account for actual holes or gaps in a scene. For example, with the same solutions currently available, new points can be added to a point cloud at locations that represent actual holes or gaps in the scene. Furthermore, some currently available solutions can add points to a point cloud that connect objects that are not connected in the scene, such as, for example, a tree top and the ground. Therefore, it would be desirable to have a method and apparatus that takes into account at least some of the problems discussed above, as well as other possible problems. SUMMARY
[0008] In an illustrative embodiment, an apparatus comprises an image processing system. The image processing system is configured to map at least a portion of points in a point cloud of a scene to a two-dimensional image of the scene to form transformed points. The image processing system is configured to create a merged data network using the two-dimensional image and the transformed points. Also, the image processing system is configured to identify new points to the point cloud using the merged data network and add the new points to the point cloud to form a new point cloud.
[0009] In another illustrative embodiment, an image processing system comprises a fusion manager, a depth value generator and a point cloud manager. The merge manager is configured to map at least a portion of points in a point cloud of a scene to a two-dimensional image of the scene to form transformed points and configured to create a merged data network using the two-dimensional image and transformed points. The merged data network includes matched elements associated with padded data vectors comprising non-zero depth values and unmatched elements associated with unfilled data vectors comprising null depth values. The depth value generator is configured to identify new depth values to replace at least a portion of the null depth values. The point cloud manager is configured to identify new points to the point cloud using the new depth values and add the new points to the point cloud to form a new point cloud.
[00010] In yet another illustrative embodiment, a computer-implemented method for increasing the number of points in a point cloud is provided. A two-dimensional image of a scene and the scene's point cloud are received. At least a portion of the points in the point cloud are mapped to the two-dimensional image to form transformed points. A merged data network is created using the two-dimensional image and the transformed points. New points for the point cloud are identified using the merged data network. The new points are added to the point cloud to form a new point cloud.
[00011] Any of the foregoing embodiments described herein may also contemplate the following computer-implemented method as described in the clauses below.
[00012] Clause 12. A computer-implemented method for increasing a number of points in a point cloud (132), the computer-implemented method comprising: receiving a two-dimensional image (121) of a scene (110) and the cloud of points (132) of the scene (110); mapping at least a portion of points in the point cloud (132) to the two-dimensional image (121) to form transformed points (146); creating a merged data network (150) using the two-dimensional image (121) and the transformed points (146); identifying new points (164) for the point cloud (132) using the merged data network (150); and add the new points (164) to the point cloud (132) to form a new point cloud (162).
[00013] Clause 13. The computer-implemented method of Clause 12, wherein mapping at least a portion of points in the point cloud (132) to the two-dimensional image (121) to form the transformed points (146) comprises: identifying the pose information for a camera system; transform a three-dimensional reference coordinate system for the point cloud (132) to a citrus three-dimensional camera coordinate system using the pose information to identify camera-centric coordinates for the points in the point cloud (132); map to at least a portion of the points in the point cloud (132) that have camera-centric coordinates to the pixel locations (124) in the two-dimensional image (121).
[00014] Clause 14. The computer-implemented method of Clause 12 or 13, wherein creating the merged data network (150) using the two-dimensional image (121) and transformed points (146) comprises: forming a data network fused (150), wherein the fused data network (150) is comprised of elements (152) that have a one-to-one correspondence with pixels in the two-dimensional image (121); eassociate the data vectors (154) with the elements (152) in the merged data network (150), wherein the data vectors (154) include padded data vectors (154) comprising non-zero depth values and depth vectors (154). unfilled data (154) comprising null depth values.
[00015] Clause 15. The computer-implemented method of Clause 14, which identifies new points (164) for the point cloud (132) using the merged data network (150) comprises: generating new depth values (158 ) to replace at least a portion of the null depth values; and identifying the new points (164) for the point cloud (132) using the new depth values (158).
[00016] Clause 16. The computer-implemented method of Clause 15, where generating the new depth values (158) to replace at least a portion of the null depth values comprises: centering a window at a location of an element in the network data array (150), wherein the element is associated with a vector of unfilled data (154) comprising a null depth value; identifying support elements (500) in a portion of the overlapping merged data network (150) through the window; and generate a new depth value (156) to replace the null depth value using support elements (500) and a linear estimation algorithm.
[00017] Clause 17. The computer-implemented method of Clause 16, wherein identifying the support elements (500) in the portion of the merged data network (150) overlapped by the window comprises: determining whether the matched elements (304, 306, 308, 310, 312, 314, 316) in the merged data network portion (150) overlaid by the window meet selected criteria; responsive to matched elements (304, 306, 308, 310, 312, 314, 316) meeting selected criteria , score each of the matched elements (304, 306, 308, 310, 312, 314, 316) with a good quality score; classify the matched elements (304, 306, 308, 310, 312, 314, 316) by score of good quality; selecting a portion of the matched elements (304, 306, 308, 310, 312, 314, 316) as the support elements (500).
[00018] Clause 18. The computer-implemented method of Clause 16 or 16, further comprising: repeating the steps of centering the window on the location of the element in the merged data network (150), identifying the supporting elements (500) in the of the merged data network (150) overlaid by the window, and generating the new depth value (156) to replace the null depth value using the support elements (500) and the linear estimation algorithm for each location in the data network castings (150).
[00019] Clause 19. The computer-implemented method of any one of clauses 15 to 18, wherein identifying the new points (164) for the point cloud (132) using the new depth values (158) comprises: identifying the camera centric coordinates within a three-dimensional camera centric coordinate system for the new points (164) using the new depth values (158); and transform camera-centric coordinates to point locations within a three-dimensional reference coordinate system for the point cloud (132) to form the new points (164).
[00020] Clause 20. The computer-implemented method of any one of clauses 15 to 19, wherein generating the new depth values (158) to replace at least a portion of the null depth values comprises: processing the data network merged (150) using windows to generate the new depth values (158).
[00021] The features and functions can be achieved independently in various embodiments of the present description or can be combined in still other embodiments in which additional details can be seen with reference to the following description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[00022] The new believed features characteristic of the illustrative embodiments are presented in the appended claims. Illustrative embodiments, however, as well as a preferred mode of use, their purposes and additional features, will be better understood by reference to the following detailed description and an illustrative embodiment of the present description when read in conjunction with the accompanying drawings, wherein:
[00023] Figure 1 is an example of an environment in which an image processing system can be used to process images and point clouds in the form of a block diagram according to an illustrative embodiment;
[00024] Figure 2 is an illustration of a merged data network according to an illustrative embodiment;
[00025] Figure 3 is an illustration of a portion of a merged data network superimposed by a window in accordance with an illustrative embodiment;
[00026] Figure 4 is an illustration of a process for generating scores for each of a group of matched elements according to an illustrative embodiment;
[00027] Figure 5 is an illustration of a selection of support elements according to an illustrative embodiment;
[00028] Figure 6 is an illustration of generating a new depth value according to an illustrative embodiment;
[00029] Figure 7 is an illustration of a merged image according to an illustrative embodiment;
[00030] Figure 8 is an illustration of a comparison between two merged images according to an illustrative embodiment;
[00031] Figure 9 is an illustration of a comparison of a final merged image according to an illustrative embodiment;
[00032] Figure 10 is an illustration of a table of final merged images according to an illustrative embodiment;
[00033] Figure 11 is an illustration of a process for increasing the number of points in a point cloud in the form of a flowchart according to an illustrative embodiment;
[00034] Figure 12 is an illustration of a process for mapping points in a point cloud to a two-dimensional image to form transformed points in the form of a flowchart according to an illustrative embodiment;
[00035] Figure 13 is an illustration of a process for creating a merged data network in the form of a flowchart according to an illustrative embodiment;
[00036] Figure 14 is an illustration of a process for generating new depth values in the form of a flowchart according to an illustrative embodiment;
[00037] Figure 15 is an illustration of a process for generating new points for a point cloud in the form of a flowchart according to an illustrative embodiment; and
[00038] Figure 16 is an illustration of a data processing system in the form of a block diagram according to an illustrative embodiment. DETAILED DESCRIPTION
[00039] The different illustrative modalities recognize and take into account different considerations. For example, the illustrative modalities recognize and take into account that it may be desirable to have a method for increasing the number of points in a point cloud by adding new points to locations in the three-dimensional reference coordinate system for the point cloud that has a level of desired precision.
[00040] Also, the illustrative modalities recognize and take into account that a two-dimensional image of the same scene captured by a point cloud can be used to increase the number of points in a point cloud. However, illustrative modalities recognize and take into account that it may be desirable to process the two-dimensional image to increase the number of points in the point cloud without making assumptions about the scene and/or the shapes of objects in the scene captured in the two-dimensional image.
[00041] Thus, illustrative modalities provide a system and method for increasing the number of points in a point cloud of a scene using a two-dimensional image of the scene. Also, this system and method can increase the number of points in the point cloud without making any assumptions about the scene.
[00042] With reference to Figure 1, an illustration of an image processing environment in the form of a block diagram is presented according to an illustrative embodiment. In Figure 1, the image processing environment 100 is an example of an environment in which the image processing system 102 can be used to process images and point clouds.
[00043] As shown, the image processing system 102 may be implemented in the computer system 104. The computer system 104 may be comprised of one or more computers and/or other types of hardware. When more than one computer is present on the computer system 104, these computers may be in communication with each other.
[00044] The image processing system 102 may be configured to receive data from a first sensor system 106 and a second sensor system 108. The first sensor system 106 and the second sensor system 108 are configured to generate data about a scene 110. Scene 110 may include features 111. Features 111 may include, for example, without limitation, objects 112. Objects 112 may include, for example, without limitation, any number of vehicles, buildings, structures made by man, people, animals, landscape features, and/or other types of objects. Further, features 111 may also include, for example, a scene background 110.
[00045] In these illustrative examples, the first sensor system 106 can take the form of the first imaging system 114. The first imaging system 114 can be any system configured to generate imaging data 116 for the scene 110. In an illustrative example, the first imaging system 114 takes the form of the camera system 118. Further, the camera system 118 may take the form of an electro-optical camera system.
[00046] Image formation data 116 may include, for example, without limitation, image 120. Specifically, image 120 may be a two-dimensional image 121. When generated by an electro-optical camera system, image 120 may be referred to as an electro-optical image.
[00047] As shown, the image 120 may be comprised of pixels 122. The pixels 122 may be arranged in a two-dimensional network comprised of rows and columns. In this illustrative example, pixel locations 124 may identify pixel locations 122 within this network. As an illustrative example, each of pixel locations 124 may identify a row and column for a corresponding pixel.
[00048] In addition, each of the pixels 122 may be associated with pixel data. The pixel data for a pixel may include, for example, without limitation a number of color values, an intensity value, and/or other values. The number of color values can include, for example, a red value, a green value, and a blue value based on the RGB color model.
[00049] In these illustrative examples, the second sensor system 108 can take the form of the second imaging system 126. The imaging system 126 can be any system configured to generate point data 128 for the scene 110. In As an illustrative example, the imaging system 126 takes the form of the light detection and range system 130.
[00050] Point data 128 may include point cloud 132. Point cloud 132 may take the form of a three-dimensional point cloud 133 in these illustrative examples. Point cloud 132 of scene 110 may be generated from a different perspective than image 120 of scene 110 in some illustrative examples.
[00051] Point cloud 132 is comprised of points 134 in a three-dimensional reference coordinate system. In these illustrative examples, each of the point locations 136 can identify the coordinates for a corresponding point in this three-dimensional reference coordinate system. In an illustrative example, the reference coordinate system may be a real world coordinate system such as, for example, a geographic coordinate system.
[00052] The resolution 140 of the point cloud 132 can be identifying the level of detail with which the features 111 in the scene 110 captured by the point cloud 132 can be discerned within the point cloud 132. In some cases, the resolution 140 of point cloud 132 may depend on the number 138 of points 134 in point cloud 132. For example, as the number 138 of points 134 in point cloud 132 increases, the resolution 140 of point cloud 132 may also increase.
[00053] The image processing system 102 is configured to receive the image 120 generated by the camera system 118 and the point cloud 132 generated by the light detection and range system 130. The image processing system 102 uses the image 120 to increase the number 138 of the point cloud 132, and thereby the resolution 140 of the point cloud 132. More specifically, the image processing system 102 can be configured to generate new points that can be added to the point cloud. 132.
[00054] As shown, the image processing system 102 may include a merge manager 142, a depth value generator 144, and a point cloud manager 145. The merge manager 142 is configured to map at least one portion of points 134 in point cloud 132 to image 120 to form transformed points 146. More specifically, merge manager 142 is configured to map point locations to at least a portion of points 134 in point cloud 132 to the pixel locations in the image plane of the image 120. The image plane of the image 120 may be the plane that lies within the two-dimensional pixel array 122.
[00055] The transformed points 146 may be formed using, for example, a transformation algorithm 148. The transformation algorithm 148 may include any number of processes, equations, and/or algorithms for mapping at least a portion of the points 134 to the pixel locations in the image plane of the image 120. In an illustrative example, the transform algorithm 148 may include a camera pose estimation algorithm such as, for example, a perspective n-point camera pose estimation algorithm efficient (EPnP).
[00056] The camera pose estimation algorithm can provide pose information for a pose of a camera system 118. The pose of the camera system 118 can be comprised of at least one of the orientation and position of the camera system 118 .
[00057] The merge manager 142 uses the transform algorithm 148 to transform the three-dimensional reference coordinate system for the point cloud 132 into a three-dimensional camera centric coordinate system. Specifically, the merge manager 142 can use the pose information for the camera system 118 provided by the camera pose estimation algorithm to transform the three-dimensional reference coordinate system into the three-dimensional camera centric coordinate system.
[00058] With this transformation, the origin of the three-dimensional reference coordinate system can be moved to the location of the camera system 118. The merge manager 142 then identifies the camera centric coordinates to the points 134 in the camera centric coordinate system 142. three-dimensional camera.
[00059] Thereafter, the merge manager 142 is configured to map the camera centric coordinates to the points 134 to the corresponding pixel locations in the image plane of the image 120 to form the transformed points 146. In this mode, the centric coordinates of camera to a point in point cloud 132 may be mapped to a pixel location that lies within image 120 or outside image 120 but on the same image plane as image 120. Transformed points 146 may include only those points that are mapped to pixel locations within the 120 image.
[00060] The merge manager 142 is configured to create a merged data network 150 using the image 120 and the transformed points 146. The merged data network 150 includes data that has been merged together based on the image 120 and the point cloud 132.
[00061] As shown, the merged data network 150 may include elements 152. Each of the elements 152 may correspond to one of the pixels 122 in the image 120, and each of the pixels 122 may correspond to one of the elements 152. Specifically, the elements 152 can have a one-to-one correspondence with the pixels 122 in the image 120. In this way, the merged data lattice 150 can be the same size as the pixel lattice 122 in the image 120.
[00062] Further, as each of the transformed points 146 corresponds to a pixel in the image 120, each of the transformed points 146 may also correspond to the element in the merged data network 150 that corresponds to that pixel. Each of the transformed points 146 can be described as mapping to a corresponding element 152.
[00063] For example, transformed points 146 may map to a first portion of elements 152. This first portion of elements may be referred to as matched elements. However, a second portion of the elements 152 may not have any transformed points that map to these elements. The second portion of elements 152 may be referred to as unmatched elements.
[00064] In these illustrative examples, each of the elements 152 may be associated with a data vector, such as, for example, data vector 154. As shown, data vector 154 may include a depth value 156. depth value 156 may be a distance between the transformed point corresponding to the element associated with the data vector 154 and the camera system 118 within the three-dimensional camera centric coordinate system.
[00065] When the element associated with the data array 154 is an unmatched element, the depth value 156 can be a null or zero value. When the element associated with the data array 154 is a matched element, the depth value 156 can be a non-null or non-zero value. The data vector 154 may be referred to as an unpadded data vector when the depth value 156 is a null or zero value and a padded data vector when the depth 156 value is a non-null or non-zero value. In this way, the unmatched elements in the fused data network 150 can be associated with the unfilled data vectors and the matched elements in the fused data network 150 can be associated with the padded data vectors.
[00066] In addition, depending on the implementation, vector data 154 may also include other data such as, for example, pixel location 157, original pixel data 159, and/or other types of data. Pixel location 157 may be the pixel location for the pixel that corresponds to the element associated with the data vector 154. The original pixel data 159 may be the pixel data in the image 120 for the pixel that corresponds to the element associated with the data vector 154.
[00067] In these illustrative examples, the depth value generator 144 is configured to generate new depth values 158 for at least a portion of the unfilled data vectors associated with the unmatched elements in the merged data network 150. Thus, depth value generator 144 may identify new depth values 158 to replace at least a portion of null depth values. The unfilled data vectors can then be filled, by the merge manager 142, with new depth values 158 to form the modified merged data network 160.
[00068] The new depth values 158 can be generated using the modified merged data network 160. Specifically, the depth value generator 144 can use windows of selected sizes to scan and process the merged data network 150. Further, techniques estimation techniques, such as linear estimation techniques, can be used to generate new depth values 158. An example of an implementation for the process of generating new depth values 158 is described below in Figures 2-6.
[00069] The point cloud manager 145 is configured to use the modified merged data network 160 to create a new point cloud 162. Specifically, the point cloud manager 145 can utilize the portion of elements in the merged data network 150 which has data vectors filled with new depth values 158 to identify new points 164 for point cloud 132.
[00070] As an illustrative example, the point cloud manager 145 can map each of the pixel locations to pixels that correspond to the portion of elements in the merged data network 150 that have data vectors filled with new depth values 158 to the camera centric coordinates in the three-dimensional camera centric coordinate system. These camera-centric coordinates can then be transformed into the three-dimensional reference coordinate system for the original point cloud, point cloud 132, to form new points 164.
[00071] This transformation may be performed using, for example, without limitation, an inverse transformation algorithm 166. The inverse transformation algorithm 166 may be the inverse of the transformation algorithm 148 used by the merge manager 142.
[00072] The point cloud manager 145 adds new points 164 to the point cloud 132 to form the new point cloud 162. This process of adding new points 164 to the point cloud 132 can be referred to as "populating" the cloud of points 132. The new point cloud 132 can have a greater number of points than the number 138 of points 134 in the point cloud 132. New points 164 can provide the new point cloud 162 with an increased resolution compared to the 140 resolution of the 132 point cloud. The new 164 points can capture the 111 features in the 110 scene with a desired level of accuracy.
[00073] Consequently, the new point cloud 162 can be used in place of the point cloud 132 to perform any number of operations. For example, without limitation, the new points 164 can improve the view of scene 110 in the new point cloud 162 compared to the point cloud 132. Also, the new points 164 can allow one or more of the objects 112 in the scene 110 are identified and/or classified using the new point cloud 162 with a higher level of accuracy compared to using the point cloud 132.
[00074] For example, the initial merged image 168 may be generated by the merge manager 142 using the image 120 and the point cloud 132. At least a portion of the points in the point cloud 132 may be mapped to pixel locations in the image 120 and superimposed on image 120 to create an initial merged image 168. The final merged image 170 may be generated by the merge manager 142 using the image 120 and the new point cloud 162. At least a portion of the points in the new point cloud 162 can be mapped to pixel locations in image 120 and superimposed on image 120 to create the final merged image 170.
[00075] The preview of scene 110 in the final merged image 170 can be improved compared to the preview of the scene 110 in the initial merged image 168. Specifically, the greater number of points in the final merged image 170 compared to the initial merged image 168 can improve the preview of scene 110 in the final merged image 170.
[00076] The illustration of the image processing environment 100 in Figure 1 is not intended to imply physical or architectural limitations on the way in which an illustrative embodiment may be implemented. Other components in addition to or in place of those illustrated may be used. Some components may be optional. Also, blocks are shown to illustrate some functional components. One or more of these blocks may be combined, divided, or combined and divided into different blocks when implemented in an illustrative embodiment.
[00077] For example, in some illustrative examples, depth value generator 144 may be considered part of merge manager 142. In other illustrative examples, merge manager 142 and/or depth value generator 144 may be considered part of the point cloud manager 145.
[00078] In other illustrative examples, some other module in the image processing system 102 may be used to form the initial merged image 168 and/or the final merged image 170. In still other cases, some other data processing system or processor unit may be configured to process image 120 and new point cloud 162 to form final merged image 170.
[00079] Looking now at Figures 2-6, illustrations of a process for generating new depth values for the unfilled data vectors associated with the unmatched elements in a fused data network are presented according to an illustrative embodiment. The process illustrated in Figures 2-6 may be an example of a way in which the new depth values 158 in Figure 1 may be generated. Further, this process can be performed using, for example, the depth value generator 144 in Figure 1.
[00080] Referring now to Figure 2, an illustration of a merged data network is presented according to an illustrative embodiment. In this illustrative example, the fused data network 200 may be an example of an implementation for the fused data network 150 in Figure 1.
[00081] In Figure 2, the merged data network 200 may be the fusion of image 202 and transformed points 206. The image 202 may be an image generated, for example, by the camera system 204. The transformed points 206 may include a portion of the points in a point cloud mapped to pixel locations in the image 202, and thereby the elements in the merged data network 200. The merged data network 200 may have been generated, for example, by the merge manager 142 in Figure 1.
[00082] As shown, depth value generator 144 can move window 108 along fused data network 200 in the direction of arrow 210 and arrow 212 to process fused data network 200. For example, window 208 can be moved to the location of each element within the merged data network 200. In this illustrative example, the window 208 can be one of a group of selected sizes. As used herein, a "group of" items may include one or more items. In this way, a group of selected sizes can include one or more selected sizes.
[00083] The merged data network 200 can be fully scanned and processed using a window that has each size in the selected size group. Moving window 208 to the location of an element in merged data network 200 means centering window 208 on that element in this illustrative example. When the element on which each window 208 is centered is an unmatched element, the portion of the merged data lattice 200 overlapped by the window 208 may be processed by the depth value generator 144 to identify a new depth value for the data array. unpopulated associated with this unmatched element.
[00084] Referring now to Figure 3 an illustration of a portion of the merged data network 200 superimposed by the window 208 of Figure 2 is presented in accordance with an illustrative embodiment. In this illustrative example, portion 300 of merged data network 200 of Figure 2 is overlapped by window 208.
[00085] In Figure 3, matched element group 302 identifies elements in portion 300 of merged data network 200 to which a portion of transformed points 206 in Figure 2 has been mapped. All other elements (not shown in this view) in portion 300 of the merged data network 200 may be unmatched elements to which transformed points have not been mapped. As shown, the matched element group 302 includes matched elements 304, 306, 308, 310, 312, 314, and 316.
[00086] Depth value generator 144 first confirms that the depth value in the data vector associated with the element on which window 208 is centered is a null depth value. If the depth value is a non-null depth value, depth value generator 144 moves window 208 to another location. If the depth value is a null depth value, depth value generator 144 continues processing portion 300 of merged data network 200.
[00087] In this illustrative example, portion 300 of merged data network 200 is shown divided into quadrants 320, 322, 324, and 326. Depth value generator 144 is configured to confirm that a sufficient number of matched elements are present. in portion 300 of merged data network 200. Further, depth value generator 144 is configured to confirm that at least one matched element is present in each of quadrants 320, 322, 324, and 326.
[00088] Once the depth value generator 144 confirms that a sufficient number of matched elements are present in the 300 portion of the merged data network 200 and that at least one matched element is present in each of the quadrants 322, 324 , and 326, depth value generator 144 may continue processing portion 300 of merged data network 200. When a sufficient number of matched elements are not present in portion 300 or when one of quadrants 320, 322, 324, and 326 is not includes a matched element, depth value generator 144 can move window 208 to a next location along merged data network 200.
[00089] Looking now at Figure 4, an illustration of a process for generating scores for each of the matched element group 302 of Figure 3 is presented in accordance with an illustrative embodiment. In Figure 4, the depth value generator 144 is configured to generate a score for each matched element in the matched element group 302. The score for a matched element may be based on the depth value in the padded data vector associated with the element. matched and a similarity of the matched element to the other matched elements in matched element group 302.
[00090] The depth value of the padded data vector associated with the matched element may be the distance between the camera system 204 and the location of the transformed point, which has been mapped to the matched element, within the centric coordinate system of three-dimensional camera. Depth values for matched element group 302 can be distances 402, 404, 406, 408, 410, 412, and 414 between matched elements 304, 306, 308, 310, 312, 314, and 316, respectively, and the 204 camera system.
[00091] The matched element's similarity to the other matched elements in the matched element group 302 can be based on any number of characteristics. These features may include, for example, without limitation, pixel location, color, intensity, and/or other types of features or data within a data vector. In this illustrative example, the similarity of the matched element to the other matched elements may be based on the distance between the pixel location identified in the data vector associated with the matched element and the ideal pixel location 400. The pixel location may be the same than the location for the matched element within the merged data network 200.
[00092] Referring now to Figure 5, an illustration of a selection of support elements is shown in accordance with an illustrative embodiment. In this illustrative example, the scores generated for the matched element group 302, as described in Figure 4, can be sorted, or ordered.
[00093] The desired number of support elements can be five support elements in this illustrative example. The five matched elements that have the lowest five scores are selected as support elements 500. Support elements 500 include matched elements 306, 308, 310, 312, and 314.
[00094] In this illustrative example, support elements 500 can be selected so that the number of new points created for actual holes and/or gaps in the scene captured by image 202 is reduced. Furthermore, the support elements 500 can be selected so that the new points created actually represent an external surface of an object.
[00095] Looking now at Figure 6, an illustration of the generation of a new depth value is presented according to an illustrative embodiment. In this illustrative example, the support elements 500 of Figure 5 can be used to generate a new depth value for the element 600 on which the window 208 has been centered. Element 600 is at location 602 in fused data network 200, which may be the central location of portion 300 of fused data network 200.
[00096] The new depth value for element 600 can be generated using, for example, a linear estimate. Specifically, a linear estimation algorithm can use the depth value in the padded data vector associated with each of the support elements 500 to estimate a new depth value for the element 600. This new depth value can be used to pad the vector data associated with element 600.
[00097] The process described in Figures 3-6 can be repeated for each location in the merged data network 200 to which the window 208 of Figure 2 is moved. Furthermore, the entire process described in Figures 2-6 can be repeated using selected windows of different sizes.
[00098] In this way, the entire merged data network 200 can be scanned and processed with a window each having a group of selected sizes. Furthermore, this entire process of using windows having the selected size group can be iterated any number of times to fill at least a portion of the unfilled data vectors associated with the unmatched elements in the merged data network 200.
[00099] The illustrations in Figures 2-6 are not intended to imply physical or architectural limitations on the way in which an illustrative embodiment may be implemented. These illustrations are presented to describe the concept of generating new depth values in an abstract way.
[000100] Referring now to Figure 7, an illustration of a merged image is presented in accordance with an illustrative embodiment. The merged image 700 may be an example of an implementation for the merged data network 150 in Figure 1. The merged image 700 may be a combination of the image 702 and transformed dots 704. In this illustrative example, each pixel in the merged image 700 may be associated with a data vector such as data vector 154 in Figure 1.
[000101] Looking now at Figure 8, an illustration of a comparison between two merged images is presented according to an illustrative embodiment. In this illustrative example, the initial merged image 800 is an example of an implementation for the initial merged image 168 in Figure 1. Further, the final merged image 802 is an example of an implementation for the final merged image 170 in Figure 1.
[000102] As shown, the initial merged image 800 is a fusion of the image 804 and the transformed points 806. The transformed points 806 may include points mapped from an original point cloud, such as, for example, the point cloud 132 in the Figure 1. The final merged image 802 is a merge of the same image 804 and transformed points 808. The transformed points 808 may include points mapped from a new point cloud, such as, for example, the new point cloud 162 in Figure 1 The scene captured in image 804 can be better realized by dots transformed 808 in final merged image 802 as compared to dots transformed 806 in initial merged image 800.
[000103] Referring now to Figure 9, an illustration of a comparison of a final merged image generated when support elements are used and a final fused image generated when support elements are not used is presented according to an illustrative embodiment . In this illustrative example, a final merged image comparison 900 and a final merged image 902 are shown. These final merged images are example implementations for the final merged image 170 in Figure 1.
[000104] The final merged image 900 is a merge of the 904 image and the transformed points 906. The final merged image 902 is a merging of the same image 904 and the transformed points 908. The transformed points 906 and the transformed points 908 can both include points mapped from a new point cloud corresponding to which new points have been added.
[000105] However, transformed points 906 may include points from a new point cloud generated without using support elements. Transformed points 908 may include points from a new point cloud generated using support elements. As shown, the surface forming the top of the building shown in portion 912 of the final merged image 900 may be less clearly defined and less accurate than the surface forming the top of the building shown in portion 914 of the final merged image 902.
[000106] Referring now to Figure 10, an illustration of a table of final merged images is presented in accordance with an illustrative embodiment. In Figure 10, table 1000 includes a column 1002, a column 1004, a row 1006 and a row 1008.
[000107] Column 1002 includes the final merged image 1010 and the final merged image 1014 generated using two iterations of scan windows that have a group of sizes selected through a network of merged data. Column 1004 includes the final merged image 1012 and the final merged image 1016 generated using ten iterations of scan windows that have a group of sizes selected through a network of merged data.
[000108] Row 1006 includes the final merged image 1010 and the final merged image 1012 generated using eight window sizes per iteration. Further, row 1008 includes the final merged image 1014 and the final merged image 1016 generated using sixteen window sizes per iteration.
[000109] As shown, the number of points included in a final merged image increases as the number of iterations and the number of sizes for the windows per iteration increase. As the number of dots in a final merged image increases, the preview of the scene within the final merged image can be improved.
[000110] The merged image illustrations in Figures 7-10 are not intended to imply physical or architectural limitations on the way in which an illustrative embodiment may be implemented. These merged images are examples of how merged images, such as the initial merged image 168 and final merged image 170 of Figure 1, can be implemented.
[000111] Referring now to Figure 11, an illustration of a process for increasing a number of points in a point cloud in the form of a flowchart is presented according to an illustrative embodiment. The process illustrated in Figure 11 can be implemented using the image processing system 102 in Figure 1.
[000112] The process begins by receiving a two-dimensional image from a first sensor system and a point cloud from a second sensor system (operation 1100). In operation 1100, the two-dimensional image may be received from a first sensor system in the form of a camera system, such as the camera system 118 in Figure 1. In this illustrative example, the two-dimensional image may be a color image. Further, the point cloud may be received from a second sensor system in the form of a light detection and range system, such as the light detection and range system 130 in Figure 1.
[000113] Both the two-dimensional image and the point cloud can be from the same scene. However, depending on the implementation, the two-dimensional image and point cloud can capture the same scene from the same or different perspectives.
[000114] Next, at least a portion of the points in the point cloud can be mapped to the two-dimensional image to form the transformed points (operation 1102). Next, a merged data network is created using the two-dimensional image and the transformed points (operation 1104).
[000115] Subsequently, new points for the point cloud are identified using the merged data network (operation 1106). The new points are added to the point cloud to form a new point cloud (operation 1108), with the process ending after this. The increased number of points in the new point cloud compared to the original point cloud can provide the new point cloud with an increased resolution compared to the original point cloud.
[000116] In other words, the new point cloud can capture features in the scene more accurately than the original point cloud. The new point cloud can be used to perform a number of different operations such as, for example, without limitation object identification, object classification, segmentation, scene visualization, and/or other types of operations.
[000117] Referring now to Figure 12, an illustration of a process for mapping the points in a point cloud to a two-dimensional image to form transformed points in the form of a flowchart is presented according to an illustrative embodiment. The process illustrated in Figure 12 can be used to implement the operation 1102 in Figure 11. Further, this process can be performed using the merge manager 142 in the image processing system 102 in Figure 1.
[000118] The process begins by transforming the three-dimensional reference coordinate system for the point cloud into a three-dimensional camera-centric coordinate system to identify the camera-centric coordinates for the points in the point cloud (operation 1200). The three-dimensional reference coordinate system can be, for example, a geographic coordinate system or some other type of real-world coordinate system. The three-dimensional camera centric coordinate system origin is the location of the camera system that generated the two-dimensional image.
[000119] For example, for each point of a given location, Xp, , , in the three-dimensional reference coordinate system, the centric camera coordinates are identified as follows:
where Xp, Yp, Zp are the coordinates for the point in the three-dimensional reference coordinate system; Xc, Yc, Zc are the camera centric coordinates for the point in the three-dimensional camera centric coordinate system; R is a rotation; and T is a translation.
[000120] The rotation, R, and the translation, T, can be identified using a transformation algorithm that includes a camera pose estimation algorithm such as an efficient perspective n-point camera pose estimation algorithm. This camera pose estimation algorithm such as an efficient perspective n-point camera pose estimation algorithm identifies the pose information for a pose of the camera system that generated the two-dimensional image. The camera system pose can be comprised of at least one of the camera system's orientation and position. The transformation algorithm uses the pose information from the camera system to generate the camera centric coordinates for the point.
[000121] Next, points that have camera-centric coordinates are mapped to pixel locations on an image plane into the two-dimensional image to form initial transform points (operation 1202). Each of the initial transformed points can be a point that corresponds to a pixel at a specific pixel location within the image plane of the two-dimensional image. For example, each point can be mapped to a pixel location, u, v, as follows:
where u is the row for pixel location and v is the column for pixel location.
[000122] After this, a portion of the initial transformed points is selected based on selected criteria to form the transformed points (operation 1204), with the process ending after this. In operation 1204, the portion of the selected initial transformed points may include points that have a row, u, that is greater than zero and less than or equal to the maximum number of rows in the two-dimensional image and that have a column, v, that is greater than zero and less than or equal to the maximum number of columns in the two-dimensional image. In this way, the transformed points can only include the pixel locations that are inside the two-dimensional image and not outside the two-dimensional image.
[000123] Referring now to Figure 13, an illustration of a process for creating a merged data network in the form of a flowchart is presented in accordance with an illustrative embodiment. The process illustrated in Figure 13 can be used to implement operation 1104 in Figure 11.
[000124] The process starts by identifying a distance for each of the transformed points (operation 1300). This distance can be the distance between the camera centric coordinates for the transformed point and the camera system. The distance can be identified as follows:
where dc is the distance.
[000125] After this, a determination is made as to whether any of the transformed points have been mapped to the same pixel location (operation 1302). If any of the transformed points were mapped to the same pixel location, then for each pixel location to which multiple transformed points were mapped, the transformed point that is closest to the camera system is kept and the other transformed points are discarded ( operation 1304).
[000126] Next, the process normalizes the depth values for each of the remaining transformed points to form normalized depth values (operation 1306). For example, for each remaining transformed point, i, the normalized distance is identified as follows:
where is the normalized distance to the transformed point; is the distance identified for the transformed point in operation 1300; is a predetermined minimum distance and is a predetermined maximum distance. The predetermined minimum distance and the predetermined maximum distance can be computed automatically using, for example, a computer system.
[000127] After this, the fused data network is created in which each element of the fused data network is associated with a data vector comprising a pixel location, a depth value, and original pixel data (operation 1308). Elements in the merged data network can have a one-to-one correspondence like pixels in the two-dimensional image. The pixel location in the data vector associated with an element in the merged data network may include the row and column for the pixel that corresponds to the element. The depth value in the data vector associated with the element can be the normalized distance identified for the transformed point that has been mapped to the pixel that corresponds to the element. If no transformed points have been mapped to the pixel that corresponds to the element, the depth value can be null. The original pixel data in the data vector associated with the element may include, for example, the red value, the green value, and the blue value for the pixel that corresponds to the element.
[000128] In this way, the data vector associated with an element in the merged data network can be represented as follows:
where qi is the data vector associated with the ith element in the merged data network; ui is the row for the pixel that corresponds to the ith element; vi is the column for the pixel that corresponds to the ith element; di is the depth value for a transformed point mapped to the pixel that corresponds to the ith element; and ri,gi,bi are the red value, the green value, and the blue value for the pixel that corresponds to the io element. When a transformed point has not been mapped to the pixel that corresponds to the element in the fused data network, the data vector associated with an element in the fused data network can be represented as follows:

[000129] Next, each of the elements in the merged data network can be indexed so that each of the elements may be able to be uniquely referenced (operation 1310), with the process ending after that. For example, each element can be indexed as follows:
where l is the index for the element and C is the number of columns in the two-dimensional image.
[000130] Elements in the merged data network that correspond to pixels to which the transformed points have been mapped are matched elements. Elements in the merged data network that correspond to pixels to which no transformed points have been mapped are unmatched elements.
[000131] Referring again to operation 1302, if none of the transformed points has been mapped to the same pixel location, the process proceeds to operation 1306 as described above. In this way, the process described in Figure 13 can be used to create a merged data network, such as the merged data network 150 in Figure 1.
[000132] Referring now to Figure 14, an illustration of a process for generating new depth values in the form of a flowchart is presented according to an illustrative embodiment. The process illustrated in Figure 14 can be used to implement operation 1106 in Figure 11.
[000133] The process starts by identifying a maximum number of iterations for processing the merged data network (operation 1400). Next, a group of selected sizes for a window is identified for use in processing the merged data network (operation 1402). After that, a size for the window is selected from the group of selected sizes (operation 1404). Each of the sizes in the selected size group can be a per size. In this way, each window can have a length and a width that are equal. In this illustrative example, each can be an odd number.
[000134] The window is moved to the location of an element in the merged data network (operation 1406). A determination is made as to whether the element is a matched element or an unmatched element (operation 1408). A matched element has an array of data with a non-null depth value. An unmatched element has a data vector with a null depth value. If the element is a matched element, a determination is made as to whether any unprocessed locations are present in the merged data network (operation 1410).
[000135] If any unprocessed locations are present in the merged data network, the process returns to operation 1406 as described above. Otherwise, a determination is made as to whether any sizes in the size group selected for the window are still present (operation 1412). If any sizes in the size group selected for the window are still present, the process returns to operation 1404 as described above. Otherwise, an iteration is now considered to be complete and a determination is made as to whether the maximum number of iterations has been reached (operation 1414). If the maximum number of iterations has not been reached, the process returns to operation 1402 as described above. Otherwise, the process creates new points for a point cloud using new generated depth values for at least a portion of the unmatched elements in the merged data network (operation 1416), with the process ending after this.
[000136] Referring again to operation 1408, if the element is an unmatched element, a determination is made as to whether the matched elements in the portion of a merged data network overlapped by the window meet selected criteria (operation 1418). Matched elements in the portion of the merged data network overlapped by the window meet the selected criteria if the number of matched elements is greater than a selected threshold and if at least one matched element is present in each quadrant of the overlapping merged data network portion through the window.
[000137] If the matched elements do not meet the selected criteria, the process proceeds to operation 1410 as described above. Otherwise, a score is generated for each of the matched elements in the portion of the merged data network overlapped by the window (operation 1420). For example, the score might be a good quality score for the matched element. The good quality score can be based on a combination of the distance from the transformed point mapped to the pixel that corresponds to the camera system element and a dissimilarity of the matched element with the other matched elements.
[000138] With matched elements in the portion of the merged data network overlapped by the window, the good quality score can be generated as follows:
where Gt is the good quality score of the ith matched element; Mt is a dissimilarity score for the ith matched element; Δt is a distance measurement for the ith matched element; j is an index for the n matched elements; F is a response to a characteristic; m is the number of features; i. is the depth value in the data vector associated with the ith matched element; and y is a normalization constant. If both characteristic responses and depth values are normalized between 0 and 1, the normalization constant, y, can be adjusted to Vm
[000139] In this illustrative example, a feature can be, for example, pixel location, intensity, color, or some other type of feature. The answer for this characteristic can be a value for this characteristic.
[000140] Next, matched elements that have a preselected number of the lowest scores are selected as supporting elements (operation 1422). For example, a number can be preselected for the desired number of support elements. This pre-selected number can be how many, five, eight, ten, or some other number. In operation 1422, the scores generated in operation 1420 can be sorted. If the preselected number is five, the matched elements that have the lowest five scores are selected as the supporting elements.
[000141] A determination is then made as to whether a support element is present in each quadrant (operation 1424). If a support element is not present in each quadrant, the process proceeds to operation 1410 as described above. Otherwise, a new depth value is generated for the element on which the window is centered using a linear estimation algorithm for the supporting elements (operation 1426).
[000142] In operation 1426, the new depth value can be identified using a linear system such as, for example: 77,
where d° is the new depth value and w0, ^ and M2 are weights. Of course, any kind of polynomial approximation algorithm can be used to solve for the new depth value using the depth values for the supporting elements.
[000143] After this, a determination is made as to whether the new depth value is within a selected range (operation 1428). The new depth value can be within the selected range ■'"VTí <-» ■
where d° is the smallest distance from a transformed point mapped to a pixel that corresponds to a support element of the camera system; wx is a window width; and a is a constant perspective adjustment weight.
[000144] If the new depth value is not within the selected range, the process proceeds to operation 1410 as described above. This way, the new depth value is not added to the data vector that corresponds to the element on which the window is centered. Instead, the depth value in this data array remains null. However, if the new depth value is within the selected range, the new depth value is added to the data vector associated with the element on which the window is centered (operation 1430). The process then proceeds to operation 1410 as described above.
[000145] With reference to Figure 15, an illustration of a process for generating new points for a point cloud in the form of a flowchart is presented according to an illustrative example. The process illustrated in Figure 15 can be used to implement operation 1416 in Figure 14.
[000146] The process starts by denormalizing the new generated depth values for at least a portion of the unmatched elements in the merged data network (operation 1500). Each of these new depth values is used to create a new point for the point cloud. The new depth values can be denormalized as follows:
where is a denormalized depth value.
[000147] Camera centric coordinates are generated for the new points for the point clouds using the denormalized depth values and the pixel locations in the data vectors for which the new depth values were generated (operation 1502). Camera centric coordinates are generated as follows:

[000148] After this, the point locations for the new points in the three-dimensional reference coordinate system in the original point cloud are identified using the camera centric coordinates (operation 1504), with the process ending after this. Point locations in the three-dimensional reference coordinate system are identified using an inverse of the transform algorithm used in operation 1200. For example, point locations can be identified as follows:
where Xp, Yp, Zp are the coordinates for the new point to be added to the point cloud in the three-dimensional reference coordinate system and Xc, Yc, Zc are the camera centric coordinates for the new point in the camera centric coordinate system three-dimensional.
[000149] The flowcharts in the different modalities presented illustrate the architecture, functionality, and operation of some possible implementations of the system and method described in the illustrative modalities. In this regard, each block in flowcharts can represent a module, a segment, a function, and/or a portion of an operation or step. For example, one or more of the blocks may be implemented using software, hardware, or a combination of the two. Hardware can, for example, take the form of integrated circuits that are manufactured or configured to perform one or more operations in the flowcharts.
[000150] In some alternative implementations of an illustrative embodiment, the function or functions noted in the blocks may occur out of the order noted in the figures. For example, in some cases, two blocks shown in succession may be executed substantially concurrently, or executed in reverse order, depending on the specific implementation. Also other blocks can be added to the blocks illustrated in a flowchart.
[000151] Looking now at Figure 16, an illustration of a data processing system in the form of a block diagram is presented according to an illustrative embodiment. The data processing system 1600 may be used to implement one or more computers in the computer system 104 in Figure 1. Further, the merge manager 142, the depth value generator 144 and/or the point cloud manager 145 in Figure 1 may be implemented using data processing system 1600. Further, a data processing system similar to data processing system 1600 may be implemented within first sensor system 106 and/or second sensor system 108 in Figure 1.
[000152] As shown, the data processing system 1600 includes a communications structure 1602, which provides communications between the processor unit 1604, the storage devices 1606, the communications unit 1608, the input/output unit 1610 and the display unit 1612. In some cases, the communications structure 1602 may be implemented as a bus system.
[000153] The 1604 processor unit is configured to execute instructions for the software to perform a number of operations. Processor unit 1604 may comprise a number of processors, a multiprocessor core, and/or some other type of processor, depending on the implementation. In some cases, the processor unit 1604 may take the form of a hardware unit, such as a circuit system, an application-specific integrated circuit (ASIC), a programmable logic device, or some other suitable type of hardware unit. .
[000154] Instructions for the operating system, applications, and/or programs executed by the processor unit 1604 may be located on storage devices 1606. Storage devices 1606 may be in communication with the processor unit 1604 through the structure 1602. As used herein, a storage device, also referred to as a computer-readable storage device, is any piece of hardware capable of storing information on a temporary and/or permanent basis. This information may include, but is not limited to, data, program code, and/or other information.
[000155] Memory 1614 and persistent storage 1616 are examples of storage devices 1606. Memory 1614 may take the form of, for example, random access memory or some type of volatile or non-volatile storage device. Persistent storage 1616 can comprise any number of components or devices. For example, persistent storage 1616 may comprise a hard disk, flash memory, rewritable optical disk, rewritable magnetic tape, or some combination of the above. The media used by 1616 persistent storage may or may not be removable.
[000156] Communications unit 1608 allows data processing system 1600 to communicate with other data processing systems and/or devices. Communications unit 1608 may provide communications using physical and/or wireless communications connections.
[000157] The input/output unit 1610 allows an input to be received from and an output to be sent to other devices connected to the data processing system 1600. For example, the input/output unit 1610 may allow an input from user is received through a keyboard, mouse and/or some other type of input device. As another example, the input/output unit 1610 may allow an output to be sent to a printer connected to the data processing system 1600.
[000158] Display 1612 is configured to display information for a user. Display 1612 may comprise, for example, without limitation, a monitor, a touch screen, a laser display, a holographic display, a virtual display device, and/or some other type of display device.
[000159] In this illustrative example, the processes of the different illustrative embodiments may be performed by the processor unit 1604 using computer-implemented instructions. These instructions may be referred to as program code, computer usable program code, or computer readable program code and may be read and executed by one or more processors in processor unit 1604.
[000160] In these examples, program code 1618 is located in functional form on computer readable media 1620, which is selectively removable, and can be loaded onto or transferred to data processing system 1600 for execution by the unit 1604. Program code 1618 and computer readable medium 1620 together form a computer program product 1622. In this illustrative example, computer readable medium 1620 can be computer readable storage medium 1624 or a computer readable storage medium 1624. computer readable signal 1626.
[000161] Computer readable storage medium 1624 is a physical or tangible storage device used to store program code 1618 rather than a medium that propagates or transmits program code 1618. Computer readable storage medium 1624 may be, for example, without limitation, an optical or magnetic disk or persistent storage device that is connected to the data processing system 1600.
[000162] Alternatively, program code 1618 may be transferred to data processing system 1600 using computer readable signal medium 1626. Computer readable signal medium 1626 may be, for example, a propagated data signal which contains program code 1618. This data signal can be an electromagnetic signal, an optical signal, and/or some other type of signal that can be transmitted over physical and/or wireless communications connections.
[000163] The illustration of the data processing system 1600 in Figure 16 is not intended to provide architectural limitations on the way in which the illustrative embodiments may be implemented. The different illustrative embodiments may be implemented in a data processing system that includes components in addition to or in place of those illustrated for the data processing system 1600. Further, the components shown in Figure 16 may vary from the illustrative examples shown.
[000164] The illustrative modalities can be implemented using any device or hardware system capable of executing program code. As an illustrative example, the data processing system may include organic components integrated with inorganic components and/or may be comprised entirely of organic components excluding a human. For example, a storage device may be comprised of an organic semiconductor.
[000165] Thus, the illustrative embodiments provide a system and method for increasing the number of points in a point cloud. In an illustrative modality, a two-dimensional image and a point cloud of the same scene are received. At least a portion of the points in the point cloud are mapped to the two-dimensional image to form transformed points. A merged data network is created using the two-dimensional image and the transformed points. New points for the point cloud are identified using the merged data network. The new points are added to the point cloud to form a new point cloud.
[000166] The new point cloud formed using the image processing system described by the illustrative modalities can allow a number of operations to be performed with a higher level of precision and/or efficiency compared to the original point cloud. For example, object identification, object classification, segmentation, and/or other image processing operations can be performed more accurately using the new point cloud compared to the original point cloud.
[000167] Also, the increased number of new points in the new point cloud can provide a better view of the scene compared to the original point cloud. Furthermore, the new point cloud can be used to better improve the two-dimensional image compared to the original point cloud.
[000168] The image processing system described by the different illustrative modalities allows this new point cloud that has an increased number of points to be formed without making any assumptions about the types of objects in the scene, about the shapes of objects in the scene, and/or on the background of the scene. In this way, the process provided by the illustrative embodiments can form a new point cloud that more accurately represents the scene as compared to a process that makes assumptions about the scene to increase the number of points in the point cloud.
[000169] The description of the different illustrative embodiments has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the embodiments as described. Many modifications and variations will be apparent to those skilled in the art. Also, different illustrative embodiments may provide different characteristics compared to other illustrative embodiments. The selected modality or modalities are chosen and described so as to better explain the principles of the modalities, the practical application, and allow others skilled in the art to understand the description for various modalities with various modifications as are suitable for the specific use contemplated.

权利要求:
Claims (3)
[0001]
1. Apparatus, comprising: an image processing system (102) configured to map a portion of points (134) in a point cloud (132) of a scene (110) to a two-dimensional image (121) of the scene ( 110) to form transformed points (146); creating a merged data network (150) using the two-dimensional image (121) and transformed points (146); wherein each of the transformed points (146) corresponds to a pixel (122) in the image (120) and each of the transformed points corresponds to an element (152) in the merged data network (150) that corresponds to that pixel (122) ; identifying new points (164) for the point cloud (132) using the merged data network (150); and adding the new points (164) to the point cloud (132) to form a new point cloud (162); a first sensor system configured to generate the two-dimensional image (121), wherein the first sensor system is a camera system, and a second sensor system configured to generate the point cloud (132), wherein the second sensor system is a light and range detection system, wherein the image processing system (102) comprises a fusion manager (142) configured to map point locations (136) to the portion of points in the point cloud (132) to pixel locations (124) in the two-dimensional image (121) to form transformed points (146) and configured to create the fused data network (150) using the two-dimensional image (121) and transformed points (146), wherein the fused data network (150) comprises matched elements (304, 306, 308, 310, 312, 314 , 316) associated with padded data vectors (154) that comprise values non-null depth values and unmatched elements associated with unfilled data vectors (154) comprising null depth values, characterized by the fact that the image processing system (102) further comprises a depth value generator (144 ) configured to identify new depth values (158) to replace at least a portion of the null depth values through the use of support elements (500) and a linear estimation algorithm, wherein support elements (500) are identified by sorting the matched elements (304, 306, 308, 310, 312, 314, 316) through a good quality score, where the good quality score is based on a combination of a distance from the transformed point (152) mapped to the pixel (122) that corresponds to the element (152) and a disparity from the matched element to the other matched elements (304, 306, 308, 310, 312, 314, 316), and selecting matched elements that have a pre-selected number of the lowest scores as supporting elements (500); wherein the image processing system (102) further comprises a point cloud manager (145) configured to identify new points (164) for the cloud of points (132) using the new depth values (158), where the point cloud manager (145) is still configured to add new points (164) to the point cloud (132) to form the new point cloud ( 162), wherein the image processing system (102) is configured to map the portion of the points in the point cloud (132) to the two-dimensional image (121) using the pose information for the camera system, wherein, with matched elements, the good quality score is generated as follows:
[0002]
2. Computer-implemented method for increasing a number of points in a point cloud (132), the computer-implemented method comprising: receiving a two-dimensional image (121) of a scene (110) and the point cloud (132) ) of the scene (110); map at least a portion of points in the point cloud (132) to the two-dimensional image (121) to form transformed points (146); create a merged data network (150) using the two-dimensional image ( 121) and the transformed points (146); wherein each of the transformed points (146) corresponds to a pixel (122) in the image (120) and each of the transformed points corresponds to an element (152) in the merged data network (150) that corresponds to that pixel (122) ;identifying new points (164) for the point cloud (132) using the merged data network (150); and adding the new points (164) to the point cloud (132) to form a new point cloud (162), wherein mapping at least a portion of the points in the point cloud (132) to the two-dimensional image (121) to forming the transformed points (146) comprises: identifying the pose information for a camera system; transforming a three-dimensional reference coordinate system for the point cloud (132) to a three-dimensional camera centric coordinate system using the information pose to identify camera centric coordinates for points in the point cloud (132); map to at least a portion of the points in the point cloud (132) that have camera-centric coordinates to the pixel locations (124) in the two-dimensional image (121); whereupon creating the merged data network (150) using the two-dimensional image (121) and transformed points (146) comprise forming a merged data network (150), wherein the merged data network (150) is comprised of elements (152) that have a one-to-one correspondence. to one with pixels in the two-dimensional image (121); eassociate the data vectors (154) with the elements (152) in the merged data network (150), wherein the data vectors (154) include padded data vectors (154) comprising non-zero depth values and depth vectors (154). unpopulated data (154) comprising null depth values, wherein identifying new points (164) for the point cloud (132) using the merged data network (150) comprises: generating new depth values (158) to replace at least a part of the null depth values; where to identify the new points (164) for the point cloud (132) using the new depth values (158), and characterized in that it generates new depth values (158) to replace at least a portion of the null depth values comprises: centering a window at a location of an element in the merged data network (150), wherein the element is associated with a vector of unfilled data (154) comprising a value r of null depth; identify support elements (500) in a portion of the merged data network (150) overlapped by the window; generate a new depth value (156) to replace the null depth value using the support elements (500 ) and a linear estimation algorithm; and wherein identifying the support elements (500) in the portion of the merged data network (150) overlapped by the window comprises: determining whether the matched elements (304, 306, 308, 310, 312, 314, 316) in the portion of the data network merged data (150) overlapped by the window meets the selected criterion, where the matched elements in the portion of the merged data network overlapped by the window meet the selected criterion if the number of matched elements is greater than a selected threshold and if at least one element match is present in each quadrant of the portion of the merged data network superimposed by the window; in response to matched elements (304, 306, 308, 310, 312, 314, 316) meeting the selected criterion, score each of the matched elements (304 , 306, 308, 310, 312, 314, 316) with a good quality score, where the quality score is based on a combination of a distance from the transformed point (146) mapped to the pixel (122) corresponding to the element ( 152) and a blast from the matched element to the other matched elements (304, 306, 308, 310, 312, 314, 316); classify the matched elements (304, 306, 308, 310, 312, 314, 316) through the good quality score ;select a portion of the matched elements (304, 306, 308, 310, 312, 314, 316) as the support elements (500), where the matched elements that have a preselected number of the lowest scores are selected as supporting elements, and where, with matched elements, the good quality score is generated as follows:
[0003]
3. Computer-implemented method, according to claim 2, characterized in that it further comprises: repeating the steps of centering the window on the location of the element in the merged data network (150), identifying the support elements (500) in the portion of the merged data network (150) overlapped by the window and generate the new depth value (156) to replace the null depth value using the support elements (500) and the linear estimation algorithm for each location in the data network. merged data (150).

类似技术:

---

公开号 | 公开日 | 专利标题

---

BR102013028714B1|2022-02-01|Apparatus and method for increasing a number of points in a point cloud

---

US9886774B2|2018-02-06|Photogrammetric methods and devices related thereto

---

KR102318023B1|2021-10-27|3-Dimensional Model Generation Using Edges

---

WO2015135323A1|2015-09-17|Camera tracking method and device

---

US9213899B2|2015-12-15|Context-aware tracking of a video object using a sparse representation framework

---

Concha et al.2014|Using superpixels in monocular SLAM

---

US20150243080A1|2015-08-27|Visual localisation

---

CN108871349B|2021-06-15|Deep space probe optical navigation pose weighting determination method

---

CN107170042B|2019-06-04|A kind of three-dimensional rebuilding method of more view Stereo matchings of unordered image

---

CN108269266A|2018-07-10|Segmentation image is generated using Markov random field optimization

---

Sun et al.2015|Line matching based on planar homography for stereo aerial images

---

Zhang et al.2015|A new high resolution depth map estimation system using stereo vision and kinect depth sensing

---

Bajramovic et al.2008|Global Uncertainty-based Selection of Relative Poses for Multi Camera Calibration.

---

EP3326156B1|2020-03-11|Consistent tessellation via topology-aware surface tracking

---

CN106688012A|2017-05-17|Depth map enhancement

---

JP2021093117A|2021-06-17|Joint detection, description system, and method

---

McCann2015|3d reconstruction from multiple images

---

El Hazzat et al.2020|Fast 3D reconstruction and modeling method based on the good choice of image pairs for modified match propagation

---

Xiao et al.2016|Image completion using belief propagation based on planar priorities

---

Zhang et al.2019|Passive 3D reconstruction based on binocular vision

---

Gao et al.2015|Local self-similarity descriptor for point-of-interest reconstruction of real-world scenes

---

Wang et al.2016|Accurate multi-view stereopsis fusing DAISY descriptor and scaled-neighbourhood patches

---

Joga et al.2019|3D reconstruction of regular objects from multiple 2D images using a reference object

---

Gadasin et al.2021|Reconstruction of a Three-Dimensional Scene from its Projections in Computer Vision Systems

---

Petran et al.2006|Stereoscopic Correspondence without Continuity Assumptions

同族专利:

---

公开号 | 公开日

---

CA2826534A1|2014-05-09|

---

JP5778237B2|2015-09-16|

---

US20140132733A1|2014-05-15|

---

JP2014096152A|2014-05-22|

---

CA2826534C|2017-06-13|

---

AU2013227985B2|2019-01-24|

---

US9811880B2|2017-11-07|

---

EP2731075B1|2018-06-20|

---

EP2731075A1|2014-05-14|

---

CN103810744B|2018-09-21|

---

KR102096673B1|2020-04-03|

---

CN103810744A|2014-05-21|

---

AU2013227985A1|2014-05-29|

---

BR102013028714A2|2014-10-07|

---

KR20140060454A|2014-05-20|

引用文献:

---

公开号 | 申请日 | 公开日 | 申请人 | 专利标题

---

---

US7187809B2|2004-06-10|2007-03-06|Sarnoff Corporation|Method and apparatus for aligning video to three-dimensional point clouds|

---

US7363157B1|2005-02-10|2008-04-22|Sarnoff Corporation|Method and apparatus for performing wide area terrain mapping|

---

US7586489B2|2005-08-01|2009-09-08|Nvidia Corporation|Method of generating surface defined by boundary of three-dimensional point cloud|

---

US7912257B2|2006-01-20|2011-03-22|3M Innovative Properties Company|Real time display of acquired 3D dental data|

---

US20080112610A1|2006-11-14|2008-05-15|S2, Inc.|System and method for 3d model generation|

---

CN101377404B|2008-07-11|2010-04-14|北京航空航天大学|Method for disambiguating space round gesture recognition ambiguity based on angle restriction|

---

US8588547B2|2008-08-05|2013-11-19|Pictometry International Corp.|Cut-line steering methods for forming a mosaic image of a geographical area|

---

WO2011013079A1|2009-07-30|2011-02-03|Primesense Ltd.|Depth mapping based on pattern matching and stereoscopic information|

---

US8885890B2|2010-05-07|2014-11-11|Microsoft Corporation|Depth map confidence filtering|

---

US20120056982A1|2010-09-08|2012-03-08|Microsoft Corporation|Depth camera based on structured light and stereo vision|

---

KR101030763B1|2010-10-01|2011-04-26|위재영|Image acquisition unit, acquisition method and associated control unit|

---

CN102129708A|2010-12-10|2011-07-20|北京邮电大学|Fast multilevel imagination and reality occlusion method at actuality enhancement environment|

---

CN103635935B|2011-03-18|2016-10-12|苹果公司|3d street|

---

US8629901B2|2011-05-19|2014-01-14|National Taiwan University|System and method of revising depth of a 3D image pair|

---

US9111444B2|2012-10-31|2015-08-18|Raytheon Company|Video and lidar target detection and tracking system and method for segmenting moving targets|US10262462B2|2014-04-18|2019-04-16|Magic Leap, Inc.|Systems and methods for augmented and virtual reality|

---

US20140363073A1|2013-06-11|2014-12-11|Microsoft Corporation|High-performance plane detection with depth camera data|

---

CN105469447A|2014-09-11|2016-04-06|富泰华工业（深圳）有限公司|Point-cloud boundary right-angle side repairing system and method|

---

US10032311B1|2014-09-29|2018-07-24|Rockwell Collins, Inc.|Synthetic image enhancing system, device, and method|

---

US9772405B2|2014-10-06|2017-09-26|The Boeing Company|Backfilling clouds of 3D coordinates|

---

FR3028988B1|2014-11-20|2018-01-19|Commissariat A L'energie Atomique Et Aux Energies Alternatives|METHOD AND APPARATUS FOR REAL-TIME ADAPTIVE FILTERING OF BURNED DISPARITY OR DEPTH IMAGES|

---

CN105701861A|2014-11-28|2016-06-22|富泰华工业（深圳）有限公司|Point cloud sampling method and system|

---

CN104484895B|2014-12-24|2018-11-20|天脉聚源（北京）科技有限公司|A kind of data display method and device|

---

CN105427239B|2015-12-05|2018-05-22|刘忠艳|A kind of two-dimensional points cloud matching process|

---

US9972067B2|2016-10-11|2018-05-15|The Boeing Company|System and method for upsampling of sparse point cloud for 3D registration|

---

US10176589B2|2017-01-31|2019-01-08|Mitsubishi Electric Research Labroatories, Inc.|Method and system for completing point clouds using planar segments|

---

US10444506B2|2017-04-03|2019-10-15|Microsoft Technology Licensing, Llc|Mixed reality measurement with peripheral tool|

---

US10816334B2|2017-12-04|2020-10-27|Microsoft Technology Licensing, Llc|Augmented reality measurement and schematic system including tool having relatively movable fiducial markers|

---

CN109978984A|2017-12-27|2019-07-05|Tcl集团股份有限公司|Face three-dimensional rebuilding method and terminal device|

---

KR102080997B1|2018-08-06|2020-02-24|현대건설주식회사|Automatic Indoor Costruction progress inspection system and methodology using 360 degree Camera|

---

US10614579B1|2018-10-10|2020-04-07|The Boeing Company|Three dimensional model generation using heterogeneous 2D and 3D sensor fusion|

---

KR20200087543A|2019-01-11|2020-07-21|삼성전자주식회사|Method and apparatus for encoding 3-dimensional data and method and apparatus for decoding 3-dimensional data|

---

KR102343123B1|2019-10-04|2021-12-27|현대건설 주식회사|Tracking management System based on photodata for construction field|

---

DE102019134324A1|2019-12-13|2021-06-17|Connaught Electronics Ltd.|A method of measuring the topography of an environment|

---

CN113409227B|2021-08-19|2021-11-30|深圳市信润富联数字科技有限公司|Point cloud picture repairing method and device, electronic equipment and storage medium|

法律状态:
2014-10-07| B03A| Publication of a patent application or of a certificate of addition of invention [chapter 3.1 patent gazette]|

---

2018-11-21| B06F| Objections, documents and/or translations needed after an examination request according [chapter 6.6 patent gazette]|

---

2020-04-07| B06U| Preliminary requirement: requests with searches performed by other patent offices: procedure suspended [chapter 6.21 patent gazette]|

---

2021-08-24| B06A| Patent application procedure suspended [chapter 6.1 patent gazette]|

---

2021-11-30| B09A| Decision: intention to grant [chapter 9.1 patent gazette]|

---

2021-12-28| B09W| Correction of the decision to grant [chapter 9.1.4 patent gazette]|Free format text: O PRESENTE PEDIDO TEVE UM PARECER DE DEFERIMENTO NOTIFICADO NA RPI NO 2656 DE30/11/2021. ATRAVES DA MENSAGEM FALE CONOSCO 988307, A REQUERENTE SOLICITA CORRIGIR O QUADRO1 DO PARECER DE DEFERIMENTO, UMA VEZ QUE NAO CONSTA A ANALISE DA PETICAO NO 870210115011 DE10/12/2021, ASSIM CONCLUO PELA RETIFICACAO DO PARECER DE DEFERIMENTO, DEVENDO INTEGRAR ACARTA PATENTE O QUE CONSTAM NO QUADRO 1 DESTE PARECER. |

---

2022-02-01| B16A| Patent or certificate of addition of invention granted [chapter 16.1 patent gazette]|Free format text: PRAZO DE VALIDADE: 20 (VINTE) ANOS CONTADOS A PARTIR DE 07/11/2013, OBSERVADAS AS CONDICOES LEGAIS. |

优先权:

---

申请号 | 申请日 | 专利标题

---

US13/673,429|US9811880B2|2012-11-09|2012-11-09|Backfilling points in a point cloud|

---

US13/673,429|2012-11-09|

---